Control interface for autonomy

ABSTRACT

A control device for an autonomous lawn mower is described which receives input signals from a first and/or second hand control and determines a control signal for controlling the autonomous lawn mower. The hand controls may provide for intuitive control of the mower by a user. The control signals may be used to operate the autonomous lawn mower to perform a task such that, when later detached or otherwise decoupled, the autonomous lawn mower may perform the same or similar tasks substantially autonomously based on data (e.g., sensor signals, control signals, etc.), generated during manual operation. In some examples, the control signals may be determined to aid a user in maintaining a straight mow, proximity to a desired pattern for mowing, and/or be otherwise altered based on the presence of a user.

BACKGROUND OF THE INVENTION

Despite advances in artificial intelligence, most autonomous systems anddevices rely on previously collected data to operate. For example,sensor data may be collected manually using a first device to be laterprocessed and used by an autonomous second device. Differences betweensuch first and second devices require expensive design changes toaccommodate manual control and, worse, may cause unintended operation ofthe computerized device based on differences between sensors and/orcontrol inputs associated with the first device and those associatedwith the second. Further, manual control of such devices may beimprecise, causing errors, while autonomous control may limit theability of an autonomous device to complete a task for complex scenariosthe autonomy is not yet capable of handling.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description will now be described, by way of examples, withreference to the accompanying drawings, where like numerals denote likeelements, a leftmost numeral indicates the original figure in which theelement is found, and in which:

FIG. 1 presents an example system of a user interfacing with anautonomous lawn mower via a control interface;

FIG. 2 illustrates an example map that may be created based on sensordata from an autonomous lawn mower, as described herein;

FIG. 3 illustrates hand controls for user input of a control interface,as described herein;

FIG. 4 depicts a user interface of a control interface, as describedherein;

FIG. 5 is an example process diagram illustrating the techniques forreceiving user input via a control interface, as described herein; and

FIG. 6 shows an example system for performing the one or more techniquesdescribed herein.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description describes techniques (e.g., methods,processes, and systems) for receiving input from a user to control adevice, such as a lawn mower, via a control interface. While connectedto the device, a user is able to directly provide control signals to thedevice while performing a task. Once the task is completed, the controlinterface may be removed (or otherwise decoupled—wirelesslydisconnected, placed into a different mode of operation, etc.). Controlsignals generated by the user and sensor signals received from sensorsassociated with the device during operation by the user may besubsequently (and/or contemporaneously) used, for example, to create amap of the area or otherwise provide information to the device to allowthe device to perform the same or additional tasks substantiallyautonomously, and/or aid the user in operation of performing a task.Such manual control of an autonomous device may, for example, ensurethat data (e.g., sensor data) received is as close as possible to datathat would have been collected from the device if operated completelyautonomously, as well as ensure that some very complex tasks which arenot possible under autonomous operation alone can be completed, therebyrendering continuous service and/or performance of a task.

As described above, autonomous systems may, in some scenarios, requireuser input to define boundaries, tasks, etc. As a non-limiting example,a user may control a lawn mower via the control input to define aboundary for future mowing. The boundary may indicate a perimeter of thearea to be mowed. In a similar example, the user may mow the entire area(whether or not in a preferred order).

To accomplish the task (e.g., of defining the boundary or performing themowing), the user may attach (or couple) the control interface to themower to provide initial controls. In some examples, this may compriseone or more of a mechanical and/or electrical coupling. Of course, theinvention is not meant to be so limiting and is only described hereinwith respect to a control interface for illustration purposes. Indeed,user input from any control device is contemplated, whether connected(mechanically and/or wired), remote (wireless, via a web/mobileinterface, or the like), and/or simply placed in a desired setting ofautonomous versus manual control.

Once the control interface is connected to the autonomous lawn mower, auser may provide one or more control inputs via the control interface.In an example, the control interface may comprise one or more inputs forreceiving controls from a user. Such inputs may be one or more ofphysical controls (e.g., knobs, dials, buttons, and the like) and/or atouch screen for both displaying information and/or receiving input fromthe user. In any of the examples described in detail herein, suchcommands may be relayed to the lawn mower via one or more of a wired orwireless communication interface.

In various examples, a user may provide a desired operating state forthe device. The operating state may indicate whether the device is to becontrolled autonomously, controlled manually, whether the device is tobe parked or disengaged, put in a neutral state, or otherwise.Additionally, or alternatively, a user may input a desired speed for themower to be propelled, a blade height for the mower, a blade speed(e.g., an RPM) for the mower, data sources for collection (a set ofsensors, control inputs, etc.), resetting of odometers, and the like.

If placed in manual operation, a user may use one or more hand controlsas input for controlling one or more of a speed or direction of themower. In some such examples, two hand controls may be provided for eachhand of an operator. In at least some examples, an input may be providedon each hand control wherein, as will be described in detail below,combinations of a set speed and hand controls from each of the handcontrol inputs may be used to control the speed (and/or torque) anddirection of the mower.

While under manual control, the mower may record sensor data from one ormore sensors onboard the mower. As non-limiting examples, such sensorsmay comprise cameras (whether RGB, monochrome, infrared, ultraviolet,etc. whether wide field of view, narrow field of view, or the like),lidar, radar, time of flight sensors, radar, ultrasonic transducers,inertial measurement units (IMUs) (which may comprise accelerometers,gyroscopes, and/or magnetometers), wheel encoders, global positioningsystems, and the like. Sensor data from the one or more sensors may berecorded as they are generated, at a particular frequency, or otherwise.In additional or alternative examples, control data input from a usermay also be recorded with a similar timestamp.

In some examples, a user may control the mower via the control interfacefor the mower to perform a task. As a non-limiting example, the task maycomprise defining a perimeter of a region to be later mowed autonomouslyby the mower. In such an example, the user may provide input to thecontrol interface to indicate an initialization of the task, may thencontrol the mower via the control interface while performing the task,and finally provide an additional input to indicate that the task hasbeen completed. In at least some examples, the signals indicative of aninitialization and/or completion of the task may be generated by aprocessor associated with one or more of the mower or the controlinterface based at least in part on connecting or decoupling the controlinterface from the device.

In various examples, limitations may be placed on the actuation of themower while the control interface is attached. As a non-limitingexample, any one or more of mower speed, blade speed, steering angle (orsteering angle rate), may be limited while controlled by a user. In atleast some examples, a sensor may detect the presence of a user, suchas, but not limited to, cameras, weight/pressure sensors, proximitysensors (such as inductive proximity sensors), etc. on a platformassociated with the user to further provide an assurance that the useris currently operating the device and/or provide differing limitations.Limiting control may ensure that the mower is safely controlled whileoperated by a user.

While operating the device, data from one or more sensors associatedwith the device, data derived therefrom, and/or control data from a userinput may be recorded for later building a map (or other datastructure). Such a map may be used by the device in order to operateautonomously over the area associated with the task.

In some examples, the one or more sensors may comprise radar, lidar(s),image sensors (RGB cameras, monochrome cameras, stereo cameras, depthcameras, infrared cameras, ultraviolet cameras, RGB-D cameras, IRcameras, etc.), inertial measurement units (IMUs), accelerometers,gyroscopes, magnetometers, global positioning systems (GPS) or GlobalNavigation Satellite System (GNSS), ultrasonic transducers (e.g.,SONAR), wheel encoders, and the like. In some examples, two or moreimage sensors may have differing fields of view (e.g., wide and narrow)and have an associated baseline. Environment features, including depthestimates, may be determined based on multi-view geometry techniques ofsensor data from the multiple image sensors and/or otherwise providedwith depth sensors provided.

Multiple mapping techniques may be used to construct the map based onthe acquired sensor data including, but not limited to SimultaneousLocalization and Mapping (SLAM), Kalman filters (Unscented KalmanFilters, Extended Kalman Filters, etc.), bundle adjustment, slidingwindow filters, occupancy grids, and the like. Such a map may be storedas a signed distance function (SDF), truncated SDF (TSDF), trianglemesh, mosaics, or other data structure. Use of voxel hashing may improvememory requirements for both storage and raycasting. In at least someexamples, control signals received from the user during mapping may beassociated with the map.

In some examples, such maps may be determined on the device.Additionally, or alternatively, sensor data from any of the one or moresensors may be stored on the device for processing on another, morepowerful, system (e.g., transmitted wired or wirelessly to a remotedevice). In such an example, memory, processing power, and weight of thedevice may be minimized.

As above, once data from the equipment is processed to determine a map,whether onboard the equipment or remote therefrom, the control inputportion may be decoupled from the equipment (whether physically removed,wirelessly decoupled, and/or whether a different mode of operation isselected (e.g., a user selecting autonomous mode on the controlinterface)). Once removed, the equipment may rely on the determined mapin order to navigate an area. In at least some examples herein, certainfunctions may be removed, limited, or otherwise altered based on whetherthe control input portion is attached or not.

Although described herein with respect to a lawn mower for purposes ofillustration, the description is not meant to be so limiting. Any of thetechniques, processes, and systems described herein may be used inconjunction with other modes of transport (including, but not limited toattached to aircraft, helicopters, multirotors, watercraft, bicycles,tractors, automobiles, motorcycles, and the like), as well as used aloneas a drone, unmanned aircraft vehicle, unmanned watercraft, or the like.Details of such a system are described in detail below with respect tothe figures.

The techniques described herein may improve the functioning of acomputing system. In such examples, user input controls may be used,either alone or in combination with sensor data associated withuser-controlled operation, to provide a basis for autonomous function ofa system, ensuring at least a locally optimal solution is obtainable forcontrolling the system to perform a specified task, while providing aninitialization framework for quickly performing such optimization.Further, such techniques may solve the technical problem of errors inautonomous operation of a system based on differences between datacaptured during such autonomous operation and data collected manuallyusing a manual system for training such a system, thereby minimizingoperating differences between the manual and autonomous systems, as wellas how to run portions (and/or all) of the autonomous pipeline on thesame input as collected during manual control (whether contemporaneouslywith such input and/or after such input has been collected).

FIG. 1 illustrates an example of a system 100 which may optionally becontrolled via a manual input from a user. In the example depicted, thesystem 100 may comprise an autonomous lawn mower 102 capable of mowingwithout input from a user. In various examples, such an autonomous lawnmower 102 may identify a region to be mowed, set a desired blade height,and proceed in accordance with a computed pattern to mow the identifiedregion.

As described above, the autonomous lawn mower 102 may require at least amodicum of prior data in order to perform the aforementioned taskautonomously and/or be unable to autonomously perform complex tasks. Asa non-limiting example, such an autonomous lawn mower 102 may, at thevery least, require a perimeter indicative of the region to mow. Inorder to provide such data, a user 116 may stand aboard a user platform108 (which, in some examples, may also be detachable) and providecontrols via a control interface 104. Though the user platform 108 isdepicted in FIG. 1 for illustrative purposes as comprising a separateunit on wheels, the disclosure is not meant to be so limiting. Ofcourse, such a user platform 108 may comprise a platform coupleddirectly to the autonomous lawn mower 102 (whether affixed ordetachable).

As will be described in further detail below, the control interface 104may have one or more input portions and/or one or more hand inputcontrols 106. In at least some examples, such a system 100 may comprisetwo hand input controls 106, a right hand input control 106A and a lefthand input control 106B, although more or fewer control inputs arecontemplated.

In various examples, the inputs to the control interface 104 maycomprise signals capable of controlling any one or more componentsand/or subcomponents of the autonomous lawn mower 102. Such controlsignals may comprise, for example, a desired speed for the autonomouslawn mower 102, a torque to be applied to either one or more of a leftwheel 114A or a right wheel 114B of the autonomous lawn mower 102(whether directly, or as determined in accordance with the techniquesdescribed herein), a braking of one or more of the left or right wheels114A, 114B, a blade height and/or blade speed for a blade 112 of themower (shown protected by a blade guard) or the like. In at least someexamples, as will be described in detail below, each hand control may beassociated with a unique wheel.

While under operation by the user 116, one or more of the controlsignals input by the user 116 via the control interface 104, sensor datafrom one or more sensors 110 disposed about the autonomous lawn mower102, and/or data derived therefrom (detections, segmentations, steeringangles, error messages, etc.) may be stored at one or more of memoryavailable at the control interface 104, the autonomous lawn mower 102,and/or a device remote therefrom.

The sensor(s) 110 may comprise one or more of lidar(s), image sensor(s)110A (which may be RGB, monochromatic, infrared, ultraviolet, etc., aswell as form the basis of stereo- or multi-view systems), radar 110C,Global Navigation Satellite System(s) (GNSS) 110B, inertial measurementunit(s) (IMU), accelerometer(s), gyroscope(s), magnetometer(s), wheelencoder(s), ultrasonic transducer(s), thermal imagers, ambient lightsensor(s), time of flight sensors, barometer(s), bolometer(s),pyrometer(s), and the like. Such sensor(s) 110 may be disposed about themowing device 102 in poses (i.e., position and/or orientations)determined to optimize a field of view. Image sensors 110A may comprisenarrow field of view cameras and/or wide-angled cameras. Multiple imagesensors 110A may be disposed about the mowing device 102 to createvarious baselines (including dual baselines). As will be described indetail below, sensor data from such sensors may both aid indetermination of vegetative health, as well as creation of thevegetative health maps.

FIG. 2 illustrates an example map 200 that may be created by any systemor method as described herein. For illustrative purposes, the map 200depicts a region in an environment, as well as a portion of the regionto be mowed by an autonomous lawn mower, such as autonomous lawn mower102. The region may be bounded on one or more sides by a boundary 210(e.g., by a street, sidewalk, source of water, or the like). As anillustrative example, a user may require only a central portion of anenvironment to be mowed autonomously by the mower. In order to definethe region, the user may control the autonomous lawn mower using thecontrol interface to capture the perimeter 220 of the area to be mowedautonomously. As described above, the user may couple the controlinterface (e.g., physically attach, wired and/or wireless couple, and/orselect a manual control on the device), send a signal indicative ofinitialization, send controls to follow perimeter 220, send a signalindicative of a termination of the task, and finally decouple thecontrol interface (whether physically, digitally (terminating aconnection, etc.), or by selecting a different mode of operation). Inadditional or alternative examples, the user may control the autonomouslawn mower to perform the mowing of the entirety of the region containedwithin perimeter 220.

As above, the map 200 may be created based on data, such as sensorand/or control data acquired during user control. Such map creation maycomprise simultaneous localization and mapping (SLAM), bundleadjustment, Kalman filters, Dense Tracking and Mapping (DTAM), or thelike which may be used to fuse data from the one or more sensors tocreate a data source representative of the area to be mowed. Additionaldetail of such techniques are provided with respect to FIG. 6 below.Regardless, the map 200 may be stored in the form of a mesh, signeddistance function (including a truncated signed distance function), ahash, a lookup, an orthonormal representation of the area, or the like.In various examples, once a map 200 is generated (either by theautonomous lawn mower or a remote computing system), the autonomous lawnmower may use the map in order to perform a further task, such as mowingthe region included in the perimeter, e.g., perimeter 220.

In those examples in which only a perimeter, e.g., perimeter 220, isprovided, the autonomous lawn mower may explore other areas during aninitialization run. During the initialization run, the autonomous mowermay assume that there are no obstacles within the perimeter 220 (e.g.,sand traps, water—lakes, ponds, etc., trees, bushes, buildings, man-madeobstacles, etc.) and attempt to explore (and/or mow) the interior regionof the perimeter 220 based on such an assumption. Motion of theautonomous lawn mower during the initialization run may be a patterndetermined based at least in part on the perimeter 220. In otherexamples, the motion may be random or otherwise to ensure that theentire region is covered. During the discovery phase of theinitialization run, if an obstacle is encountered (e.g., as may bedetected by sensor data), the autonomous mower may attempt tocircumnavigate the obstacle before continuing to explore (whether mowingor moving). Once circumnavigation is complete, the mower may thencontinue to explore by moving (and/or mowing) in the predefined pattern.Sensor and/or control data may be acquired during the initialization runto supplement the map 200 with data over the entire region containedwithin perimeter 220, as well as to provide any indications of obstaclesdiscovered.

In those examples in which the user performed a mow of the entire regioncontained by perimeter 220 and those in which control data was storedduring an initialization run, the control data may be associated withthe map 200. As a non-limiting example, a state of the autonomous lawnmower (position, orientation, velocity, etc.) may be associated with thelocation on the map corresponding to the position, as well asinformation associated with torques, steering angles, blade heights,blade speeds, etc. In such an example, the autonomous lawn mower may usethe control data associated with the map in order to optimize a patternfor mowing or control signal. As a non-limiting example, such controldata may be indicative of, for example, torques applied to the one ormore wheels, blade heights, blade speeds, etc. In such examples, controldata comprising torques, speeds, mower states, etc., may be used as areference during an optimization of the lawn mower pattern or controlsto one or more of ensure the autonomous mower uses a minimum amount ofenergy, mows according to a user defined input, provides aninitialization value for control input (which may aid, for example, indetermining torques when going up/down inclines) or the like. In atleast some examples, blade heights, blade speeds, etc., associated withthe map 200 may be used when mowing autonomously to ensure a consistencyin the cut of grass between being mowed by a user and mowedautonomously. In at least some such examples, independent input from theone or more hand controls may be associated with the map 200 to be usedin such optimizations.

In determining controls for performing the mowing, one or more of theautonomous lawn mower and/or the a computing system remote from theautonomous lawn mower may rely on topographical data, weather data, timeof year data, and the like in determining a preferred pattern, such aspattern 230, to be followed when performing the mow. In variousexamples, pattern 230 may comprise a series of waypoints indicative of adesired velocity, position, blade height, blade velocity, wheel torque(for the one or more wheels), etc.

FIG. 3 depicts a detailed view of an example control interface 300(which may be control interface 104) including one or more handcontrols, such as hand control 304. Hand control(s) 304 may be used toprovide input for controlling the autonomous lawn mower during operationby a user. Each of the one or more hand control(s) 304 may comprise agrip 306 to facilitate grasping onto the hand control(s) 304 by a user.Such a grip 306 may, in some examples, enable a user to safely hold onand maintain control of an autonomous lawn mower throughout operation,despite any accelerations which may be present. In at least someexamples, the grip 306 may comprise a touch sensor (e.g., any one ormore of a capacitive, resistive, inductive sensor, switch, button, etc.)to determine whether the user is present.

To facilitate the operation of the autonomous lawn mower, controls maybe embedded in one or more of the hand control(s) 304. In the exampleillustrated, the controls may comprise an analog joystick 308, orotherwise a one-dimensional joystick. In the example provided, theanalog joysticks(s) 308 (which may be one-dimensional, two-dimensional,etc., and, in some examples, may comprise an analog rocker, a Halleffect angle sensor, a rotary encoder, a linear encoder, or the like)are embedded in the grip 306 such that a user may easily provide inputwhile holding on to the grip 306. Here, the analog joystick 308 may moveforward and backwards along direction 310. In at least some examples,such an analog joystick 308 may, by default, remain in a neutralposition halfway between a maximum forward and maximum backwarddirection. In additional or alternative examples, the grip 306 may haveone or more additional input buttons, including, but not limited to, abutton which sends a signal when analog joystick 308 is depressed intothe drip 306.

Each of the one or more hand control(s) 304 may be affixed to, orotherwise coupled to (whether electrically, mechanically, wired, orwirelessly) a portion of the control interface 312 having one or moreadditional input and/or output regions.

As will be described in detail below with respect to FIG. 4, signalsgenerated from the one or more controls associated with the handcontrol(s) 304 (such as signals from the analog joystick 308) may beused to provide a desired torque, velocity, etc., to the one or morewheels associated with the autonomous lawn mower. In some such examples,one or more processors located in the control interface 300 and/or oneor more processors in the autonomous lawn mower may limit, or otherwisealter, the controls input by a user via the control interface 300 whencoupled to the autonomous lawn mower. In one such example, when a useris detected operating the hand controls (e.g., when the controls areattached (via an electrical coupling, wireless coupling, or otherwise),based on a change of a user platform such as, but not limited to, aproximity sensor (such as an inductive proximity sensor or inductivecoupling on the user platform, a pressure sensor on the user platform,etc.), a detection of the user using cameras on board the autonomouslawn mower, etc.), touch sensor associated with the hand controls, etc.,input controls may be altered to minimize, for example, a maximum speed,a maximum rotation rate, a maximum torque, a minimum blade height, amaximum blade speed, or the like, as a user may not be able to safelyand/or reliably operate the autonomous lawn mower in such regimes. Insimilar such examples, such limitations may be relaxed when the controlinterface 300 is decoupled from the autonomous lawn mower.

In further such examples, processors on one or more of the controlinterface 300 and/or the autonomous lawn mower may alter user providedinput to the control interface 300 to ensure, for example, consistentmowing and safe operation of the autonomous lawn mower. As non-limitingexamples, control signals may be determined based on user input to causethe autonomous lawn mower to move along straight lines, to performsmooth transitions between blade speeds and/or heights, to ensureconsistent and uniform turns, to limit proximity to obstacles, tofollow/track an object (such as a curb or fence line—either detected insensor data, determined from map data associated with a map of the area,or the like), to ensure that the autonomous lawn mower stays within apredefined boundary, and the like. In at least some examples where apattern, such as pattern 230 illustrated in FIG. 2, is available to theautonomous lawn mower when the control interface is engaged, the one ormore processors on the control interface 300 and/or the autonomous lawnmower may ensure positional deviations from the pattern are minimized(e.g., by limiting a control input proportionally to a Euclidiandistance to a nearest point along the pattern), while relaxing any oneor more other constraints. As a non-limiting example, a user may be ableto control the autonomous lawn mower at speeds faster or slower, atblade heights higher or lower, at blade speeds faster or slower, etc.,than those associated with the pattern, so long as the autonomous lawnmower proceeds along the pattern. In those examples in which an objectis tracked or followed, sensor data from the one or more sensorsassociated with the autonomous lawn mower may be used to ensure theautonomous lawn mower does not run into the object and/or off the curb,while in the cases of fences and/or curbs, a consistent distance ismaintained to the tracked object.

In some examples, the user provided input may be evaluated to determineif the provided controls are sufficient to perform a given maneuver. Asa non-limiting example, if a user is attempting to climb a hill with theautonomous lawn mower while in manual control, the resultant torquedetermined based at least in part on the input from the one or more handcontrols may be compared against an amount of torque required to mountthe hill (whether determined based on previous controls used by theautonomous lawn mower to mount the hill and associated with map data,based on motor feedback, forward progression, and/or kinematic and/ordynamic models of the autonomous lawn mower, and the like), additionaltorque may be added to the torque indicated by the user to compensate.In a further such example, torque may be applied to one side (or wheel)to compensate for various orientations of the autonomous lawn mowerwhile advancing up the hill. Of course, any other parameter whichcontrols an aspect of the autonomous lawn mower is contemplated (e.g.,blade speed, blade height, etc.).

In at least some examples where a pattern is to be followed, a singlehand control may be used which may comprise a one-, two-, ormulti-axis/input joystick or input interface. In such examples, a usermay control the autonomous lawn mower one-handed by providing anindication of the speed with one axis of the joystick and, in thosewhere more than one axis is used, additional controls with the otheraxis (blade speed, blade height, orientation, etc.). In similarexamples, velocity may be provided by a single hand control, where adirection (forward or reverse) may be provided via the other handcontrol. In such examples, the autonomous lawn mower may ensure that theuser is following a desired mowing pattern, but allow the user tocontrol the speed and direction of such mowing. In at least some suchexamples, a user may not need to continuously provide input via the oneor more hand controls. As a non-limiting example, the autonomous lawnmower may proceed according to a most recent control. As an example, auser may indicate to proceed forward upon which the autonomous lawnmower may follow a pattern until the user provides an additional signalto stop, slow down, reverse, or the like.

Though depicted in FIG. 3 as a control interface 300 to be connectedwith separate hand control inputs 304, this is for illustration purposesonly and the invention is not meant to be so limiting. Of course, insome examples, the control interface 300 may comprise a remote controlnot connected to the autonomous lawn mower, a web- ormobile-application, and/or have differing interface for hand inputcontrols.

FIG. 4 depicts a user interface 400 for a control interface as describedin detail herein. Although depicted in FIG. 4 as having a combination ofphysical inputs (e.g., buttons, knobs, switches, etc.), for illustrationpurposes, the invention is not meant to be so limiting. Indeed, any oneor more of the control inputs described herein may be replaced, removed,or otherwise represented in another input form such as, but not limitedto, a touch screen interface, a web-based or mobile application forinput, or the like.

As pictured in FIG. 4, the user interface 400 may comprise a screen 410.The screen 410 may display information to a user/operator indicative ofinformation associated with the autonomous lawn mower to which the userinterface 400 is connected (whether mechanically, electrically, wired,or wirelessly, etc.). Such information may comprise, for example, stateinformation of the autonomous lawn mower (e.g., position, velocity,acceleration, orientation, blade speed, blade height, etc.), statusinformation associated with the autonomous lawn mower (e.g., batterystate of charge, temperature of one or more components/subcomponents, aset or desired velocity, a set or desired blade speed, a set or desiredblade height, a set or desired task state (e.g., defining a perimeter,starting mowing, starting task, finishing task, finished mowing,finished perimeter, etc.), etc.), as well as operating state of theautonomous lawn mower (e.g., autonomous mode, manual mode, park, drive,etc.). In at least some examples, the screen 410 may comprise a “touchscreen” such that a user may provide input directly via the screen 410(for any input described herein, as authentication mechanisms, etc.).Where the screen 410 is not a touch screen, the user interface may haveone or more buttons, such as cursor control(s) 412 which allow a user toinput information to the user interface 400. Additionally, oralternatively, the user interface 400 may comprise a selectable knob 414such that the user can selectively provide input for a desired state ofoperation of the autonomous lawn mower (e.g., any one or more ofautonomous mode, manual mode, parked, neutral, etc.), such that a usermay input a desired state of the autonomous lawn mower and/or a velocityknob 416 such that a user may input a desired speed from a minimum speedto a maximum speed. Of course, any more or fewer inputs or input regionsare contemplated, whether through menus, submenus, via web-based/mobileapplications (which may be used in conjunction with the user interface400 and/or as the user interface 400). Of course, though displayed onthe user interface 400 for illustrative purposes, the disclosure is notmeant to be so limiting. In fact, any of the inputs provided withrespect to FIG. 4 may be provided in one or more of the hand controls tofacilitate user operation and control of the autonomous lawn mower whileoperated manually.

In at least some examples, one or more of the screen 410 and/or otherfeedback devices (speakers, vibrating actuators, etc.), may be used toprovide the user with additional information. In a non-limiting example,at least a portion of sensor data from one or more sensors associatedwith the autonomous lawn mower (e.g., any one or more of cameras,lidars, radars, ultrasonics, and the like) may be used to determine thepresence of an obstacle (whether static—trees, shrubs, fallen branches,man-made obstacles, or the like-, or dynamic—animals, pedestrians,etc.), determine the proximity to the obstacle, and perform one or moreactions (such as stopping the autonomous lawn mower) and/or alerting theuser to the obstacle via one or more of the screen 410 and/or any otheroutput (speakers, vibrating actuators, etc.). Alerting the user may bebeneficial as, at least in some instances, the user may not be able tosee from the vantage point of standing behind the autonomous lawn mower.In any such examples, obstacle detections determined from the sensordata may be relayed to a central server and made accessible to others toensure the obstacle is cleared for the current and/or future runs, aswell as to inspect the current mowing for safe operation.

In one example, controls received from the one or more hand controls(not pictured) may be proportional to torques provided to the respectivewheels. In such examples, when no input is received from a hand controlinput (e.g., when an analog rocker-style joystick is used and is in theneutral position), no torque may be applied to the corresponding wheel(e.g., when a right hand control rocker is in the neutral position, thecorresponding right wheel may have no torque applied to it). Similarly,when such a rocker is moved forward or backwards, torque is applied tothe corresponding wheel to cause the wheel to move forwards orbackwards, respectively. Such torque may scale linearly or non-linearlywith respect to the input hand controls (or user specified wheelvelocity). In those examples in which a maximum speed or velocity is set(e.g., via the velocity knob 416), the torque applied may be scaled(whether linearly or non-linearly) relative to the maximum velocityinput such that a corresponding linear velocity induced by rotation ofeither wheel does not exceed the maximum velocity.

In additional or alternative examples, the controls may be augmented bythe user input velocity in what may be referred to as a “push-to-steer”control. In such examples, a user may input a desired velocity (orspeed) via the velocity input knob 416. The autonomous lawn mower, insuch examples, may apply a torque to the one or more wheels to cause thewheels to move forward at the user input velocity. Pushing forward on arocker-style joystick in such an example causes a reduction in speed ofthe opposing wheel. As a non-limiting example, pushing forward on ajoystick associated with a right hand control input will cause torqueand/or velocity of a left wheel to be reduced. In any example herein,such speed/velocity reduction may be a combination of one or more ofreversing a direction of current to the motor, reducing an amount ofcurrent to the motor, and/or applying a braking force (e.g., by abrake). The reduction of the speed or torque of the opposing wheel maybe scaled, whether linearly or non-linearly, such that pushing therocker-style joystick fully forward may cause the opposing wheel to stopcompletely. Further, in such an example, pulling the rocker-stylejoystick backwards may cause a wheel on the same side as the handcontrol in which the input was provided to rotate backwards. A maximumvelocity and/or torque of the wheel may be limited based on the velocityinput from the velocity knob 416. As a non-limiting example, pulling theright hand rocker backwards will cause the right wheel to spinbackwards. In any such example, the rotation rate, torque, or velocitymay be scaled, whether linearly or non-linearly, from zero (or whateverpositive velocity is set based on, for example, the velocity knob 416)to the maximum negative velocity.

In any of the examples described herein, one or more transitions may beused in order to smooth changes between a push-to-steer when operatingin a forward direction and/or a “additive” operation when operating in abackwards direction. In such examples, the controls may be divided intoseveral regimes and a smoothing function may be applied to ensure thatcontrols vary smoothly from one regime to the other. As a non-limitingexample, a sigmoid function may be used to transition the controls fromone regime to the other.

While described above with respect to hand controls being associatedwith one or more torques to apply (or reductions in speed to be applied)to one or more wheels associated with the autonomous lawn mower,additional data may be used to determine such control signals. In anyone or more of the examples provided herein, sensor data (including, butnot limited to inertial measurement units (IMUs), cameras, lidar, radar,ultrasonics, wheel encoders, global positioning/navigation systems, andthe like) may be used to determine a state of the autonomous lawn mower.Such a state may comprise, for example, any one or more of a position,orientation, speed, velocity (whether linear or rotational),acceleration (whether lateral or longitudinal), jerk, and the like. Insuch examples, the control signals may be determined based at least inpart on the current state of the autonomous lawn mower and the one ormore input signals from the one or more hand control inputs. Inparticular in such examples, the control signals may be determined inorder to effectuate a body-centric command That is to say in thoseexamples each hand control does not directly control a wheel of theautonomous lawn mower, but rather a control is determined based on theinput and relative to the current motion of the autonomous lawn mower asa whole. As a non-limiting example, the user input(s) may be relative tothe current state of the autonomous lawn mower and control signals maybe determined accordingly. Here, the hand control inputs are not tied toa particular wheel, per se, but the inputs are associated with motion ofone wheel or another based at least in part on the current state of theautonomous lawn mower. In one such example, a linear body velocity maybe defined as the average between the detected ground velocity for eachof a starboard (right) and port (left) wheel, scaled by the throttle (orselected velocity). A maximum angular body velocity may be defined asthe sum of the port and starboard ground velocities scaled inversely bythe wheel separation and directly proportional to the throttle and ascale factor (e.g., 0.6, or any number between 0-1). The desired linearbody velocity may be determined by multiplying the current linearvelocity by one (1) plus the push-to-steer contributions of the leftand/or right hand controls (e.g., as may be determined in accordancewith the details above). Similarly, the desired angular body velocitymay be determined based on the maximum angular velocity scaled by thedifference in the push-to-steer angular contributions of the port andstarboard wheels. As illustrated in the above example, the controlsignals may be based on the current state and/or the user inputs andeach hand control input may still be associated with a wheel of theautonomous lawn mower to effectuate the input. Of course, such controlsignals may further be determined in accordance with any othertechniques disclosed herein, including, but not limited to causing thedevice to maintain a straight line, causing the device to follow/trackan object, causing the device to follow a predefined pattern, or thelike, as well as limited based at least in part on any user preferences,such as those described with respect to FIG. 5 below.

FIG. 5 illustrates an example flow diagram representing one or more ofthe processes as described herein. Each block of the flow diagram mayrepresent a module of code to execute and/or combinations of hardwareand/or software configured to perform one or more processes describedherein. Though illustrated in a particular order, the following figuresare not meant to be so limiting. Any number of blocks may proceed in anyorder (including being omitted) and/or substantially simultaneously(i.e., within technical tolerances of processors, etc.), to perform theoperations described herein.

FIG. 5 illustrates an example 500 for receiving user input via a controlinterface and generating a control signal based on the user input inorder to control an autonomous lawn mower in accordance with any of thedescriptions herein.

At operation 502, a control device may be coupled to an autonomous lawnmower. Such coupling may comprise one or more of a mechanical coupling,an electrical coupling, a selection of a mode of operation, and/or awireless coupling (whether as a remote control, by a web-basedapplication, mobile application, or similar) In those examples in whicha wireless coupling is used, the control device may be a remote controlusing any one or more communication interfaces as described with respectto FIG. 6 and/or a web application or mobile application.

At operation 504, a user presence may be determined. A user presencesignal may be generated, for example, based on an proximity sensor(e.g., inductive proximity sensor or inductive coupling) and/or pressuresensor associated with a user platform associated with the autonomouslawn mower, a touch sensor (e.g., any one or more of an inductive,capacitive, or resistive sensor) associated with the one or more handcontrols, and/or receive an RFID or other wireless authentication. Inthose examples in which a wireless authentication is provided (e.g.,where a user swipes an RFID), the autonomous lawn mower may also receiveor determine user preferences for operation such as, but not limited to,a desired maximum speed, minimum speed, blade height, blade speed,maximum torque, aggressiveness with which the autonomous lawn mowerrounds curves (e.g., lateral accelerations), or the like. In at leastsome examples, the user presence may also be indicative of a signalreceived from a user to start a task. In some examples, user presencemay be determined based at least in part on input provided by a usersuch as, but not limited to, pressing a button (e.g., a push to start),performing an ordered combination of actions (button presses or otherinput), selecting a particular operating mode for the device, and thelike. The user presence may also be associated with a specific user,such as a user having an associated unique identifier. Such a user IDmay be accessible to the autonomous lawn mower via the RFID, imagerecognition, or otherwise. In at least some examples, the user may berequired to authenticate (e.g., using a password, tapping the RFID code,etc.), before the system will allow additional input.

In either operations 506 or 508, hand controls signals may be providedby either one or more of a first or second hand control associated withthe control interface. In at least some examples, the hand control maybe based at least on a percentage forward or backward a rocker (orjoystick, etc.), is on the one or more hand controls as described in anyof FIGS. 3-4 or otherwise herein. Of course, in at least some examples,such hand controls may comprise only one such control such that a usermay control the device with a single hand.

Additionally, or alternatively, at 510, a user may interact withadditional inputs to create an additional signal. The additional signal(or signals) may be indicative of one or more parameters associated withthe autonomous lawn mower. In some examples, such signals may comprise amaximum speed, a minimum speed, a maximum torque, maximum or minimumblade height, maximum or minimum blade speed, and/or any otherconfigurable parameter associated with the autonomous lawn mower.Further, though depicted in FIG. 5 as being in parallel with 506 and508, such additional signals may be generated before receiving suchfirst and second signals.

At 512, data may be received and be based on (and/or indicative of), forexample, a current state (position and/or orientation, velocity,rotational velocity, etc.), of the autonomous lawn mower which, in someexamples, may be relative to a desired state. In at least some examples,the data may reflect a difference in the desired state and currentmeasured state of the autonomous lawn mower. The additional datareceived at 512 may further comprise, in some examples, map data sensordata from one or more sensors associated with the autonomous lawn mower,localization information (which may be determined based at least in parton one or more of the map data and/or the sensor data), pattern data, orthe like.

At 514, and as described in detail above, one or more control signalsmay be generated based at least in part on the first signal, secondsignal, the additional signal, and/or the data.

At 516 the control signals may be transmitted to a controller (actuator)associated with the autonomous lawn mower to cause the autonomous lawnmower to operate in accordance with the signal.

At 518, it may be determined whether the task (mowing, defining aperimeter, etc.), has been completed. Such a determination may be basedon, for example, a signal received from the user indicating completion,by determining that a pattern has been traced, determining that an areahas been fully covered (and/or cut to a desired height), and the like.If the task is complete, flow proceeds to 520 wherein the device isdecoupled. Otherwise, the flow may return to 504 to ensure that the useris still present. In at least some examples in which a unique user ID isreceived or determined, additional information may be transmitted toanother device (e.g., a remote server) and/or determined with respect tothe user after the device is decoupled in 520 and/or the task iscompleted. As non-limiting examples, a system (whether locally at theautonomous lawn mower via the control interface, at a server remote fromthe autonomous lawn mower, etc.), may determine statistics regarding thespecific user. Such statistics may comprise, for example, an amount ofarea mowed, a time mowed, an average speed, an average grass height,etc. The statistics may then be made available (e.g., via the internet)to any one or more other users (e.g., company representatives) orapplications for further use or processing.

Once the device is decoupled in 520, the autonomous lawn mower may mowthe same and/or similar area again without the user input (i.e., theautonomous lawn mower may mow autonomously).

Example System

FIG. 6 is an example system 600 capable of performing the operationsdescribed herein. Such a system 600 may comprise one or more ofprocessors 602, memory 604, sensor(s) 612, communication subsystem 614,actuators 616, and power system 618. Further, though depicted in FIG. 6as a single system 600 for illustrative purposes, the intention is notto be so limiting. For example, the system 600 may be a distributedsystem (either locally or non-locally), where each block may be presenton (or performed by) a remote system. Further, though particular blocksare associated with individual systems or subsystems, the disclosure isnot meant to be so limiting. Indeed, any block may be present in any oneor more of the systems or subsystems illustrated in FIG. 6 (or notpresent at all).

The system 600 may include one or more processors 602, any of whichcapable of performing the operations described herein. In some examples,the processor(s) 602 may be located remotely from the system 600. Theone or more processor(s) 602 may comprise one or more central processingunits (CPUs), one or more graphics processing units (GPUs), digitalsignal processors (DSPs), field programmable gate arrays (FPGAs),application specific integrated circuits (ASICs), or the like.

Memory 604 is an example of non-transitory computer readable mediacapable of storing instructions which, when executed by any of the oneor more processor(s) 602, cause the one or more processor(s) 602 toperform any one or more of the operations described herein (e.g., thosedescribed in reference to any of FIGS. 1-6). The memory 604 can store anoperating system and one or more software applications, instructions,programs, and/or data to implement the methods described herein and thefunctions attributed to the various systems. In various implementations,the memory 604 can be implemented using any suitable memory technology,such as static random access memory (SRAM), synchronous dynamic RAM(SDRAM), nonvolatile/Flash-type memory, or any other type of memorycapable of storing information. The architectures, systems, andindividual elements described herein can include many other logical,programmatic, and physical components, of which those shown in theaccompanying figures are merely examples that are related to thediscussion herein. Additionally, or alternatively, the memory 604 iscapable of storing raw sensor data from the one or more sensor(s) 612,compressed or downsampled sensor data, output (or intermediaterepresentations) of one or more machine learning models (e.g., featuremaps of neural networks), and/or representations of the raw sensor data.

Sensor(s) 612 may comprise one or more image sensor(s, radar(s),lidar(s), ultrasonic(s), touch sensors, Global Positioning and/orNavigation Satellite Systems, inertial measurement units (IMUs) —whichmay comprise one or more accelerometers, gyroscopes, and/ormagnetometers, and the like. Image sensors may comprise, for example,RGB cameras, intensity cameras (e.g., greyscale or monochrome), stereocameras, depth cameras (e.g., structured light sensors, time of flight(TOF) cameras, etc.), RGB-D cameras, infrared cameras, ultravioletcameras, hyperspectral cameras, and the like. In those examples wheremultiple image sensors are contemplated, various image sensors may havevarying fields of view. For example, where at least two image sensorsare used, one image sensor may be a narrow field of view camera and theother a wide angle field of view camera.

Sensor(s) 612 may further include, for example, encoders (such as wheelencoders), ultrasonic transducers (e.g., SONAR), thermal imaging sensors(e.g., infrared imagers), non-contact temperature sensors (e.g., sensorscapable of determining the temperature of a surface), ambient lightsensors (e.g., light sensors such as, but not limited to, photodiodescapable of determining an intensity of light at 600-1200 nm), humiditysensors, pressure sensors, bolometers, pyrometers, wind speed sensors,and the like. Sensor data from such other sensors 612 may be used togenerate the three-dimensional maps and/or localize the device 600. Anyof the one or more sensor(s) 612 may also be associated with a timestampincluding, but not limited to, a time of day, time of month, and/or timeof year (e.g., 1-16-2018 4:50 am UTC). In some examples, a user mayspecify whether to update an existing map and/or generate a new map.

Such an example system 600 as shown in FIG. 6 may additionally oralternatively comprise one or more communication subsystems 614. Anexample communication subsystem 614 may be used to send and receive dataeither over a wired or wireless communication protocol, as well asprovide data connectivity between any one or more of the processors 602,memory 604, and sensors 612. Such protocols may include, but are notlimited to, WiFi (802.11), Bluetooth, Zigbee, Universal Serial Bus(USB), Ethernet, TCP/IP, serial communication, cellular transmission(e.g., 4G, 5G, CDMA, etc.), and the like. As indicated herein, such acommunication subsystem 614 may be used to send data (e.g., sensor data,control signals, etc.), to other systems (e.g., cloud based computers,etc.). In at least some examples, to minimize an amount of datatransferred (as raw sensor data may amount to upwards of multiplegigabytes to multiple terabytes per day), raw sensor data from the oneor more sensors 612 may be downsampled or compressed beforetransmission. In at least one example, sensor data (whether raw,compressed, downsampled, a representation thereof, or otherwise) may beautomatically uploaded to another computing device when in a particularlocation (e.g., when in a shed, or other preselected user location).Representations of data may include, for example, averages of the data,feature maps as output from one or more neural networks, extractedfeatures of the data, bounding boxes, segmented data, and the like.

The system 600 may comprise actuator(s) 616, such as, but not limitedto, one or more motors to provide torque to one or more wheelsassociated with the system 600, a linear actuator to raise and lower ablade platform, a motor to spin a blade for cutting, one or more brakesassociated with the one or more wheels, and the like. Such actuators mayfurther comprise, for example, electric and/or mechanical motors,hydraulics, pneumatics, and the like. Upon receiving a signal from oneor more of the planning and control subsystem 610, at least a portion ofthe actuator(s) may actuate in order to effectuate a trajectory(steering, acceleration, etc.), release fertilizer, seed, herbicide,pesticide, insecticide, seed, etc., and the like.

System 600 may also comprise a power system 618 including, but notlimited to batteries, super capacitors, or otherwise to provide power tothe one or more processor(s) 602, actuators 616, sensor(s) 612, or anyother component or subcomponent of the system 600 which requires power.

Within memory 604, a calibration component 606 may perform calibrationof the one or more sensor(s) 612 and/or actuators 616. Calibration maycomprise determining one or more sensor intrinsics and/or extrinsics, aswell as determining positions of components or subcomponents (e.g.,blade height), applied torques relative to currents applied, and thelike. Such calibration protocols performed by calibration component 606may ensure that any one or more components or subcomponents of system600 is working properly and enable correct calculations to be generatedgiven the system's 600 current understanding of the relative positions,orientations, and parameters of the other components and subcomponents.

A mapping and localization subsystem 608 may take in sensor data fromany one or more of the sensor(s) 612, in addition to any one or moreoutputs from the calibration subsystem 606 to one or more of map an areaand/or provide a position and/or orientation of the system 600 relativeto the map. In at least one example, sensor data from the one or moresensor(s) 612 may be used to construct (and/or update) a two- and/orthree-dimensional map of the scanned area. When updating, preexistingmap data may be received from memory 604 and/or from server 630.Multiple mapping techniques may be used to construct a two- orthree-dimensional map based on the acquired sensor data including, butnot limited to SLAM, Kalman filters (Unscented Kalman Filters, ExtendedKalman Filters, etc.), occupancy grids, bundle adjustment, slidingwindow filters, and the like. Such a map may be stored as a signeddistance function (SDF), or truncated SDF (TSDF), triangle mesh,mosaics, etc. Use of voxel hashing may improve memory requirements forboth storage and raycasting. In at least some examples, sensor data mayinclude radar data indicative of subterranean objects (e.g., pipes, golfballs, rocks, etc.). Such subterranean objects may provide features foruse in creating the map. For example, locations of sprinklers, piping,rocks, moisture levels, and the like may be combined (or fused) withother sensor data to both generate the maps and localize against them.

Furthermore, various combinations of sensor data may be used to provideadditional insight as derived sensor data. As a non-limiting example,sensor data from wide-angle, dual baseline, image sensors may be used toreconstruct depth of the environment and provide additional features foruse in generating the map and or localizing the system 600 against sucha map. Any such derived sensor data may be either used for mappingand/or localization, as well as may be associated with the map after ithas been generated (e.g., storing the value associated with the portionof the map where the data was collected). Further, in at least someexamples, control signals (as may be received and/or generated by system600) may be associated with the map at mapping and localizationcomponent 608. In some examples, GNSS data may be used to inform aRegion of Interest (ROI) of satellite imagery to download to, orotherwise augment, the two- or three-dimensional map. Additionally, oralternatively, such a system 600 may download, or otherwise access,weather data as additional sensor data. The weather data may beindicative of, for example, weather conditions for the time of dayassociated with the other sensor data.

Such maps may comprise signed distance functions (SDFs) or truncatedsigned distance functions TSDFs, mesh representations, UTM grids,mosaics, tiles, etc., including any topological relationship betweensuch sensor data. In some examples, voxel hashing may be used tominimize memory requirements for both map storage and retrieval. Such amap may also be associated with additional sensor data (and/or dataderived from the additional sensor data, such as segmentations,classifications, output from machine learning algorithms, etc.). Forexample, moisture level data, soil density data, vegetative healthindicators (growth, absence of growth, presence of pests, presence ofweeds or invasive species, etc.), thermal data, ambient light data,etc., may be associated with every location in the three-dimensionalmap. Additionally, or alternatively, image sensor data (e.g., color) maybe associated with the map as well (e.g., by weighted averaging, or thelike), so that a user viewing the map would quickly see a virtualrepresentation of the scanned area, including color.

The planning and control subsystem 610 may determine commands foroperating one or more of the actuator(s) 616. In some examples, such aplanning and control subsystem 610 may determine one or moretrajectories for the system 600 to follow (e.g., by determining a seriesof steering commands, acceleration commands, etc., which cause thesystem 600 to follow an intended pattern). Such trajectories may bedetermined in accordance with waypoints (e.g., GNSS-based waypoints) asmay be received from a user via control interface 620 and/or calculatedto optimize (e.g., minimize) a length of travel over a defined region ofinterest (e.g., as may be determined by server 630, such as by thepattern generation component 638). Such calculations may be determined,for example, using Bellman Ford's algorithm, Dijkstra's Algorithm, orotherwise. In those examples in which a control interface (controlinterface 620) is coupled to the system 600, the signals received fromthe control interface 620 may be directly applied to the actuators 616and/or used to determine resultant control signals to apply to theactuators 616 in accordance with the techniques set forth herein (e.g.,to cause a user to follow a pattern, mow in a straight line, etc.), asdescribed in further detail in FIGS. 1-5.

In those examples in which the system 600 is operated manually (e.g., bybeing pulled, pushed, attached to a mower, etc.), the planning andcontrol subsystem 610 may nonetheless calculate an optimal controlpattern for a user to take. As a non-limiting example, such a trajectorymay comprise the shortest pattern needed to sweep an area (region)provided and/or otherwise determine control signals for actuation basedon input received from a user in order to perform a particular task.

In any such example provided herein, such trajectories and/or controlsmay be calculated iteratively (and/or periodically) such that the system600 (and/or associated user(s)) always has the most relevantinformation.

Control Interface

The system 600 may receive input from a user via a control interface620. The control interface 620 may comprise one or more processor(s) 622and/or memory 624, which may be similar to processor(s) 602 and memory604. In additional or alternate, examples, the control interface 620 maycomprise various input and/or output interface(s) I/O 628. I/O 628 maycomprise one or more hand controls and/or additional inputs, asdescribed in detail in any of FIGS. 1-4, as well as input to receive anRFID, authentication, touch sensors, or otherwise. Memory 624 maycomprise limit controls 626. Such limit controls 626 may limit theinputs provided via the I/O 628 (either based on whether a user isdetected at the system 600 and/or to determine one or more controlsignals based on the input received from the I/O 628 as describedherein. The control interface 620 may be coupled to system 600 viainterface 619. As described in detail above, such an interface 619 maycomprise one or more of a mechanical coupling (latches, hooks, livinghinges, tension fittings, fasteners, and the like), electrical couplings(e.g., to provide one or more of power to and/or communications betweenthe system 600 and the control interface 620), and or a wirelesscoupling. In those examples in which a wireless coupling is used, such awireless coupling may use a similar communication protocol as describedwith respect to communication component 614 and/or communicate via theinternet when a mobile and/or web application (e.g., as relayed by aserver 630) to system 600.

Server

Server 630 may comprise one or more processor(s) 632, memory 634, and/ora communication component 640. The processor(s) 632 and memory 634 maybe the same or similar to processor(s) 602 and memory 604. Similarly,communication component 640 may be similar to communication component614. Though not illustrated for clarity, server 630 may communicate witheither of the system 600 and/or control interface 620 via thecommunication component 640. Further, though depicted as a singleentity, the server 630 may comprise a distributed system, whetherlocally or remote from one another.

Memory 634 may comprise a mapping component 636. Such a mappingcomponent 636 may receive sensor data (and/or data derived therefrom)from the one or more sensor(s) 612 (e.g., via communication component640) and use such sensor data to generate a map of an area. Mappingcomponent 636 may be similar to, or the same as, mapping/localizationcomponent 608.

Memory 634 may further comprise a pattern generation component 638. Sucha component may determine an optimal path and/or waypoints for thesystem 600 to follow to perform a given task such as, but not limitedto, a path to follow when mowing an indicated area.

Though not illustrated for clarity, server 630 may further receive anyother form of data (e.g., user authentication, obstacle detections, andthe like) and further process the data and make it available to anyother user. As several non-limiting examples, server 630 may computestatistics about a user operating the system 600 (such as amount of timemowed, an area mowed, and the like) and make that available to an enduser (e.g., via a web interface). Similarly, additional data aboutobstacles received from the system 600 and/or control interface 620 maybe sent to an additional user to ensure that the obstacle is cleared forlater autonomous mowing by the system 600.

Of course, though described above as three distinct systems (system 600,control interface 620, and server 630), any one or more of thecomponents or subcomponents may be located in any one or more of theother systems or subsystems and any process or operation describedherein may be performed on any one or more system or subsystem(including in in a distributed fashion).

EXAMPLE CLAUSES

A. A control device for interfacing with an autonomous lawn mower, thecontrol device comprising: a first hand control configured to generate afirst signal; a second hand control configured to generate a secondsignal; one or more additional input regions to generate an additionalinput signal; one or more processors; and one or more non-transitorycomputer readable media storing instructions thereon which, whenexecuted by the one or more processors, cause the one or more processorsto perform the operations comprising: receiving sensor data from one ormore sensors associated with the autonomous lawn mower; determining,based at least in part on the sensor data and one or more of the firstsignal, the second signal, or the additional signal, a control signal;and controlling the autonomous lawn mower based at least in part on thecontrol signal.

B. The control device as described in example clause A, wherein thesensor comprises one or more of an inertial measurement unit (IMU), andwherein determining the control signal comprises: determining, based atleast in part on the sensor data, a current state of the autonomous lawnmower, the current state comprising one or more of a position, anorientation, a linear velocity, or an angular velocity; determining,based at least in part on one or more of the first signal, the secondsignal, or the additional signal, a desired state of the autonomous lawnmower; and determining, based at least in part on a difference betweenthe current state and the desired state, the control signal, wherein thecontrol signal comprises an amount of torque to be applied to a wheelassociated with the autonomous lawn mower.

C. The control device as described in either of example clauses A or B,wherein the control device is coupled to the autonomous lawn mower viaone or more of: a mechanical coupling, an electrical coupling, or awireless coupling.

D. The control device as described in any of example clauses A-C,wherein the first signal received from the first hand control of thecontrol interface is associated with a wheel speed of a first wheel ofthe autonomous lawn mower and the second signal received from the secondhand control interface is associated with a wheel speed of a secondwheel of the autonomous lawn mower, and wherein the additional input isindicative of a maximum speed of the autonomous lawn mower.

E. The control device as described in any of example clauses A-D,wherein determining the control signal comprises: receiving a currentposition of the autonomous lawn mower; determining a distance betweenthe current position and a closest point along a pattern to be mowed bythe autonomous lawn mower while operated by the user; and determining,based at least in part on the distance, the first signal, and the secondsignal, a torque to apply to a wheel associated with the autonomous lawnmower.

F. The control device as described in example clauses A-E, wherein theoperations further comprise determining a user presence signal based atleast in part on one or more of: the additional signal, a signalreceived from an RFID tag, an inductive proximity sensor associated witha platform on which the user is standing, a pressure sensor associatedwith the platform, a capacitive sensor associated with one or more ofthe first or second hand controls, an inductive sensor associated withone or more of the first or second hand controls, a resistive sensorassociated with one or more of the first or second hand controls, orcamera data from one or more cameras associated with the autonomous lawnmower, and wherein determining the control signal is further based atleast in part on the user presence signal.

G. A method for controlling an autonomous lawn mower comprising:receiving, from a control device, a signal associated with a handcontrol of the control device; receiving, from the control device, anadditional input; receiving sensor data from a sensor associated withthe autonomous lawn mower; determining, based at least in part on thesensor data, a current state of the autonomous lawn mower, the currentstate indicative of one or more of a position, an orientation, or avelocity of the autonomous lawn mower; determining, based at least inpart on the current state of the autonomous lawn mower and one or moreof the signal or the additional signal, a control signal; andcontrolling the autonomous lawn mower based at least in part on thecontrol signal.

H. The method as described in example clause G, wherein the controldevice is coupled to the autonomous lawn mower via one or more of: amechanical coupling, an electrical coupling, or a wireless coupling.

I. The method as described in any one of example clauses G or H, themethod further comprising determining whether a user is associated withthe control device based at least in part on one or more of: aninductive proximity sensor associated with a platform on which the useris standing, camera data from a camera associated with the autonomouslawn mower, a pressure sensor, an RFID tag, or the additional input, andwherein determining the control signal is further based at least in parton determining whether a user is associated with the control device.

J. The method as described in any one of example clauses G-I, whereinthe control signal comprises one or more of: an amount of torque toapply to a wheel of the autonomous lawn mower on a same side associatedwith the hand control, an amount of a reduction in speed to apply to anopposing wheel of the autonomous vehicle on an opposite side associatedwith the hand control, or an output of a piecewise smooth functioncomprising at least a sigmoid function.

K. The method as described in any one of example clauses G-J, whereinthe current state of the autonomous lawn mower comprises a position, andwherein determining the control signal comprises: receiving a patternalong which the autonomous lawn mower is to mow; determining a distancebetween the position and a closest point on the pattern; anddetermining, as the control signal and based at least in part on thedistance, a torque to apply to a wheel associated with the autonomouslawn mower.

L. The method as described in any one of example clauses G-K, furthercomprising receiving map data associated with an environment throughwhich the autonomous lawn mower is traversing, wherein determining thecontrol signal is further based at least in part on the map data.

M. The method as described in any one of example clauses G-L, whereinthe signal is generated at a first time, and wherein the sensor data isreceived at a second time after the first time, the method furthercomprising determining, at a third time after the second time, theautonomous lawn mower to be operated in an autonomous mode, wherein,after the third time, the autonomous lawn mower is configured to mowexclusive of additional signals generated by the control device.

N. One or more non-transitory computer readable media havinginstructions stored thereon which, when executed by one or moreprocessors, cause the one or more processors to perform the operationscomprising: receiving, from a control device, a signal associated with ahand control of the control device; receiving, from the control device,an additional input; receiving sensor data associated with a sensor ofan autonomous lawn mower coupled to the control device; determining,based at least in part on the sensor data and one or more of the signalor the additional input, a control signal, wherein the control signal isconfigured to control an autonomous lawn mower.

O. The one or more non-transitory computer readable media described inexample clause N, wherein the hand control comprises a joystick and thesignal is determined based at least in part on an orientation of thejoystick relative to a neutral position of the joystick.

P. The one or more non-transitory computer readable media described ineither example clause N or O, wherein the operations further comprisedetermining, based at least in part on the sensor data, whether a useris associated with the control device, and wherein determining thecontrol signal is further based at least in part on determining whetherthe user is associated with the control device.

Q. The one or more non-transitory computer readable media as describedin any of example clauses N-P, wherein determining the control signalcomprises: determining, based at least in part on the sensor data, acurrent state of the autonomous lawn mower, the current state comprisingone or more of a position, an orientation, or a velocity of theautonomous lawn mower; determining, based at least in part on one ormore of the signal or the additional input, a desired state of theautonomous lawn mower; determining a difference between the currentstate and the desired state; and determining, as the control signal andbased at least in part on the difference, an amount of torque to applyto a wheel of the autonomous lawn mower.

R. The one or more non-transitory computer readable media as describedin any of example clauses N-Q, wherein determining the control signalcomprises: determining, based at least in part on the sensor data, acurrent position of the autonomous lawn mower; receiving a pattern alongwhich the autonomous lawn mower is to mow; determining a distancebetween the current position and a closest point on the pattern; anddetermining, as the control signal and based at least in part on thedistance, a torque to apply to a wheel associated with the autonomouslawn mower.

S. The one or more non-transitory computer readable media as describedin any of example clauses N-R, wherein the operations further comprisereceiving map data, and wherein the control signal is further based atleast in part on the map data.

T. The one or more non-transitory computer readable media as describedin any of example clauses N-S wherein the signal is received at a firsttime, wherein the sensor data is received at a second time after thefirst time, and wherein, after the second time, the autonomous lawnmower is configured to mow a pattern based at least in part onadditional sensor data from the one or more sensors and exclusive ofadditional signals generated by the control device.

CONCLUSION

Here multiple examples have been given to illustrate various featuresand are not intended to be so limiting. Any one or more of the featuresmay not be limited to the particular examples presented herein,regardless of any order, combination, or connections described. In fact,it should be understood that any combination of the features and/orelements described by way of example above are contemplated, includingany variation or modification which is not enumerated, but capable ofachieving the same. Unless otherwise stated, any one or more of thefeatures may be combined in any order.

As above, figures are presented herein for illustrative purposes and arenot meant to impose any structural limitations, unless otherwisespecified. Various modifications to any of the structures shown in thefigures are contemplated to be within the scope of the inventionpresented herein. The invention is not intended to be limited to anyscope of claim language.

Where “coupling” or “connection” is used, unless otherwise specified, nolimitation is implied that the coupling or connection be restricted to aphysical coupling or connection and, instead, should be read to includecommunicative couplings, including wireless transmissions and protocols.

Any block, step, module, or otherwise described herein may represent oneor more instructions which can be stored on a non-transitory computerreadable media as software and/or performed by hardware. Any such block,module, step, or otherwise can be performed by various software and/orhardware combinations in a manner which may be automated, including theuse of specialized hardware designed to achieve such a purpose. Asabove, any number of blocks, steps, or modules may be performed in anyorder or not at all, including substantially simultaneously, i.e.,within tolerances of the systems executing the block, step, or module.

Where conditional language is used, including, but not limited to,“can,” “could,” “may” or “might,” it should be understood that theassociated features or elements are not required. As such, whereconditional language is used, the elements and/or features should beunderstood as being optionally present in at least some examples, andnot necessarily conditioned upon anything, unless otherwise specified.

Where lists are enumerated in the alternative or conjunctive (e.g., oneor more of A, B, and/or C), unless stated otherwise, it is understood toinclude one or more of each element, including any one or morecombinations of any number of the enumerated elements (e.g., A, AB, AB,ABC, ABB, etc.). When “and/or” is used, it should be understood that theelements may be joined in the alternative or conjunctive.

What is claimed is:
 1. A control device for interfacing with anautonomous lawn mower, the control device comprising: a first handcontrol configured to generate a first signal; a second hand controlconfigured to generate a second signal; one or more additional inputregions to generate an additional input signal; one or more processors;and one or more non-transitory computer readable media storinginstructions thereon which, when executed by the one or more processors,cause the one or more processors to perform the operations comprising:receiving sensor data from one or more sensors associated with theautonomous lawn mower; determining, based at least in part on the sensordata and one or more of the first signal, the second signal, or theadditional signal, a control signal; and controlling the autonomous lawnmower based at least in part on the control signal.
 2. The controldevice of claim 1, wherein the sensor comprises one or more of aninertial measurement unit (IMU), and wherein determining the controlsignal comprises: determining, based at least in part on the sensordata, a current state of the autonomous lawn mower, the current statecomprising one or more of a position, an orientation, a linear velocity,or an angular velocity; determining, based at least in part on one ormore of the first signal, the second signal, or the additional signal, adesired state of the autonomous lawn mower; and determining, based atleast in part on a difference between the current state and the desiredstate, the control signal, wherein the control signal comprises anamount of torque to be applied to a wheel associated with the autonomouslawn mower.
 3. The control device of claim 1, wherein the control deviceis coupled to the autonomous lawn mower via one or more of: a mechanicalcoupling, an electrical coupling, or a wireless coupling.
 4. The controldevice of claim 1, wherein the first signal received from the first handcontrol of the control interface is associated with a wheel speed of afirst wheel of the autonomous lawn mower and the second signal receivedfrom the second hand control interface is associated with a wheel speedof a second wheel of the autonomous lawn mower, and wherein theadditional input is indicative of a maximum speed of the autonomous lawnmower.
 5. The control device of claim 1, wherein determining the controlsignal comprises: receiving a current position of the autonomous lawnmower; determining a distance between the current position and a closestpoint along a pattern to be mowed by the autonomous lawn mower whileoperated by the user; and determining, based at least in part on thedistance, the first signal, and the second signal, a torque to apply toa wheel associated with the autonomous lawn mower.
 6. The control deviceof claim 1, wherein the operations further comprise determining a userpresence signal based at least in part on one or more of: the additionalsignal, a signal received from an RFID tag, an inductive proximitysensor associated with a platform on which the user is standing, apressure sensor associated with the platform, a capacitive sensorassociated with one or more of the first or second hand controls, aninductive sensor associated with one or more of the first or second handcontrols, a resistive sensor associated with one or more of the first orsecond hand controls, or camera data from one or more cameras associatedwith the autonomous lawn mower, and wherein determining the controlsignal is further based at least in part on the user presence signal. 7.A method for controlling an autonomous lawn mower comprising: receiving,from a control device, a signal associated with a hand control of thecontrol device; receiving, from the control device, an additional input;receiving sensor data from a sensor associated with the autonomous lawnmower; determining, based at least in part on the sensor data, a currentstate of the autonomous lawn mower, the current state indicative of oneor more of a position, an orientation, or a velocity of the autonomouslawn mower; determining, based at least in part on the current state ofthe autonomous lawn mower and one or more of the signal or theadditional signal, a control signal; and controlling the autonomous lawnmower based at least in part on the control signal.
 8. The method ofclaim 7, wherein the control device is coupled to the autonomous lawnmower via one or more of: a mechanical coupling, an electrical coupling,or a wireless coupling.
 9. The method of claim 7, the method furthercomprising determining whether a user is associated with the controldevice based at least in part on one or more of: an inductive proximitysensor associated with a platform on which the user is standing, cameradata from a camera associated with the autonomous lawn mower, a pressuresensor, an RFID tag, or the additional input, and wherein determiningthe control signal is further based at least in part on determiningwhether a user is associated with the control device.
 10. The method ofclaim 7, wherein the control signal comprises one or more of: an amountof torque to apply to a wheel of the autonomous lawn mower on a sameside associated with the hand control, an amount of a reduction in speedto apply to an opposing wheel of the autonomous vehicle on an oppositeside associated with the hand control, or an output of a piecewisesmooth function comprising at least a sigmoid function.
 11. The methodof claim 7, wherein the current state of the autonomous lawn mowercomprises a position, and wherein determining the control signalcomprises: receiving a pattern along which the autonomous lawn mower isto mow; determining a distance between the position and a closest pointon the pattern; and determining, as the control signal and based atleast in part on the distance, a torque to apply to a wheel associatedwith the autonomous lawn mower.
 12. The method of claim 11 furthercomprising receiving map data associated with an environment throughwhich the autonomous lawn mower is traversing, wherein determining thecontrol signal is further based at least in part on the map data. 13.The method of claim 7, wherein the signal is generated at a first time,and wherein the sensor data is received at a second time after the firsttime, the method further comprising determining, at a third time afterthe second time, the autonomous lawn mower to be operated in anautonomous mode, wherein, after the third time, the autonomous lawnmower is configured to mow exclusive of additional signals generated bythe control device.
 14. One or more non-transitory computer readablemedia having instructions stored thereon which, when executed by one ormore processors, cause the one or more processors to perform theoperations comprising: receiving, from a control device, a signalassociated with a hand control of the control device; receiving, fromthe control device, an additional input; receiving sensor dataassociated with a sensor of an autonomous lawn mower coupled to thecontrol device; determining, based at least in part on the sensor dataand one or more of the signal or the additional input, a control signal,wherein the control signal is configured to control an autonomous lawnmower.
 15. The one or more non-transitory computer readable media ofclaim 14, wherein the hand control comprises a joystick and the signalis determined based at least in part on an orientation of the joystickrelative to a neutral position of the joystick.
 16. The one or morenon-transitory computer readable media of claim 14, wherein theoperations further comprise determining, based at least in part on thesensor data, whether a user is associated with the control device, andwherein determining the control signal is further based at least in parton determining whether the user is associated with the control device.17. The one or more non-transitory computer readable media of claim 16,wherein determining the control signal comprises: determining, based atleast in part on the sensor data, a current state of the autonomous lawnmower, the current state comprising one or more of a position, anorientation, or a velocity of the autonomous lawn mower; determining,based at least in part on one or more of the signal or the additionalinput, a desired state of the autonomous lawn mower; determining adifference between the current state and the desired state; anddetermining, as the control signal and based at least in part on thedifference, an amount of torque to apply to a wheel of the autonomouslawn mower.
 18. The one or more non-transitory computer readable mediaof claim 14, wherein determining the control signal comprises:determining, based at least in part on the sensor data, a currentposition of the autonomous lawn mower; receiving a pattern along whichthe autonomous lawn mower is to mow; determining a distance between thecurrent position and a closest point on the pattern; and determining, asthe control signal and based at least in part on the distance, a torqueto apply to a wheel associated with the autonomous lawn mower.
 19. Theone or more non-transitory computer readable media of claim 18, whereinthe operations further comprise receiving map data, and wherein thecontrol signal is further based at least in part on the map data. 20.The one or more non-transitory computer readable media of claim 14,wherein the signal is received at a first time, wherein the sensor datais received at a second time after the first time, and wherein, afterthe second time, the autonomous lawn mower is configured to mow apattern based at least in part on additional sensor data from the one ormore sensors and exclusive of additional signals generated by thecontrol device.